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Title: Photo-realistic talking-heads from image samples 

Author: Cosatto, Eric; Graf, Hans Peter 

Corporate Source: AT&T Lab-Research, Red Bank, NJ, USA 

Source: IEEE Transactions on Multimedia v 2 n 3 Sep 2000. p 152-163 

Publication Year: 2000 

CODEN: ITMUF8 ISSN: 1520-9210 

Language: English 

Document Type: JA; {Journal Article) Treatment: T; (Theoretical) 
Journal Announcement: 0012W1 

Abstract: This paper describes a system for creating a photo-realistic 
model of the human head that can be animated and lip-synched from 
phonetic transcripts of text. Combined with a state-of-the-art 
text-to-speech synthesizer (TTS) , it generates video animations of 
talking heads that closely resemble real people. To obtain a naturally 
looking head, we choose a Mata-driven' approach. We record a talking 
person and apply image recognition to extract automatically bitmaps of 
facial parts. These bitmaps are normalized and parameterized before being 
entered into a database. For synthesis, the TTS provides the audio track, 
as well as the phonetic transcript from which trajectories in the space of 
parameterized bitmaps are computed for all facial parts. Sampling these 
trajectories and retrieving the corresponding bitmaps from the database 
produces animated facial parts. These facial parts are then projected and 
blended onto an image of the whole head using its pose information. This 
talking head model can produce new, never recorded speech of the person who 
was originally recorded. Talking-head animations of this type are useful 
as a front-end for agents and avatars in multimedia applications such 
as virtual operators, virtual announcers, help desks, educational, and 
expert systems. ( Author abstract) 43 Refs. 

Descriptors: Multimedia systems; Pattern recognition; Animation ; Speech 
synthesis; Sampling; Speech analysis; Computer vision 

Identifiers: Photo-realistic talking-heads; Text-to-speech synthesizer; 
Sample based image synthesis 
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723.5 (Computer Applications) ; 741.1 ( Light /Optics ) ; 751.5 (Speech); 
922.1 (Probability Theory) ; 741.2 (Vision) 

723 (Computer Software); 741 (Optics & Optical Devices); 751 
(Acoustics); 922 (Statistical Methods) 
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Title: Explanatory lifelike avatars: Performing user-centered tasks in 
3D learning environments 

Author: Lester, James C; Zettlemoyer , Luke S.; Gregoire, Joel P.; Bares, 
William H. 

Corporate Source: North Carolina State Univ, Raleigh, NC, USA 
Conference Title: Proceedings of the 1999 3rd International Conference on 
Autonomous Agents 

Conference Location: Seattle, WA, USA Conference Date: 

19990501-19990505 

Sponsor: ACM SIGART 

E.I. Conference No,: 55456 

Source: Proceedings of the Interantional Conference on Autonomous Agents 
1999. p 24-31 

Publication Year: 1999 
CODEN: 002624 
Language: English 

Document Type: JA; (Journal Article) Treatment: A; (Applications); T; 
(Theoretical) 

Journal Announcement: 9911W3 

Abstx-act : Because of their multimodal communicative abilities and strong 
visual presence, animated pedagogical agents offer significant promise 
for 3D learning environments. We describe a new class of animated 
pedagogical agents , explanatory lifelike avatars , which can perform 
user-designed tasks in rich 3D worlds. By generating task networks to 
perform student-designed tasks, an avatar task planner constructs and 
interprets action specifications that it then interprets within the 
geometries of the 3D environment to generate navigational, 
manipulative, and verbal behaviors. Filmed by a narrative camera planner in 
the 3D world, the avatars perform students* tasks and accompanies them 
with running verbal explanations in realtime. The explanatory lifelike 
avatar framework has been implemented in a full-scale avatar for the CPU 
CITY learning environment, a 3D learning environment for the domain of 
computer architecture and systems for novices. To investigate the 
effectiveness of this approach, a novel four-way comparative usability 
study was conducted with an ^ agentless ' world, a disembodied narrator, a 
mute lifelike avatar , and a full-scale explanatory avatar . Results of 
the study suggest that explanatory lifelike avatars hold much promise for 
learning environments. ( Author abstract) 20 Refs. 

Descriptors: ^Neural networks; Artificial intelligence; Learning systems; 
Computer architecture; Knowledge based systems 

Identifiers: Explanatory lifelike avatars; Lifelike agents; Pedagogical 
agents; Synthetic agents 

Classification Codes: 

723.4.1 (Expert Systems) 

723.4 (Artificial Intelligence) 

723 (Computer Software) ; 722 (Computer Hardware) 
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Monitoring the constructed self: Avatars, our on-line representations 

Author: Rampoldi-Hnilo, Lynn Ann 
Degree: Ph.D. 
Year: 1999 

Corporate Source/Institution: Michigan State University (0128) 
Adviser: Bradley S. Greenberg 

Source: VOLUME 60/10-A OF DISSERTATION ABSTRACTS INTERNATIONAL. 

PAGE 3568. 133 PAGES 
Descriptors: MASS COMMUNICATIONS / PSYCHOLOGY, PERSONALITY 
Descriptor Codes: 0708; 0625 

Avatars are graphical on-line images that individuals create 
to represent themselves in virtual social interactions. In this digital 
form , individuals can define all aspects of their physical appearance 
without morphology constraints. Research suggests that individuals 
differentially monitor their self-presentations (Snyder, 1974, 1979), with 
some using external cues to create self -presentations (high self -monitors ) 
and others integrating their own internal aspects (low self-monitors). This 
study was designed to test these concepts with avatars as the new 
self-presentation form . Specifically , self-presentation, sensitivity to 
expressive behaviors, other-directedness and social comparison were 
examined in relation to individuals' created avatars across three 
contexts. Individuals (N = 169) completed two on-line surveys and 
participated in a repeated measures design, with individuals creating one 
avatar for each of three contexts— to interact in an on-line chatroom 
with either: a group of friends; strangers; or a future employer. The first 
survey consisted of background questions and the individual trait measures. 
After participants had created their avatars, they answered questions 
regarding corollary dimensions of self-monitoring that they encoded into 
their self-presentations. A content analysis was conducted of the 507 
avatars to determine the types and frequency of nonverbal presentation 
elements. Will individuals who monitor themselves highly in their 
self-presentations in real-life also monitor their on-line representations 
and present themselves distinctly in different on-line contexts? Findings 
suggest that individuals did not differ their types of portrayal, 
demographics, or presentation characteristics of their avatars across 
situations by self -monitoring dimensions. In general, participants reported 
encoding more internal cues (attitudes and feelings), less external cues 
(environmental based) and more cross-situation variability in their avatar 
representations across situations. For the most part, there were few 
findings related specifically to the monitoring dimensions: 
self-presentation, sensitivity to expressive behaviors, other-directedness 
and social comparison. 

Due. to the lack of prediction with these subscales in this new 
environment, it is necessary to consider the validity of the 
self-monitoring construct. This author suggests greater explication of the 
sensitivity to expressive behavior measure— which was the most 
predictive— and reintegration of the acting dimension. 
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RECONCEIVING THE COMPUTER GAME AS AN INSTRUMENT FOR EXPRESSION: 
NARRATIVE, CONTEXT, AND CONTENT IN AtODIO DESIGN FOR MULTIMODAL SYSTEMS 

Author: BACK, MARIBETH JOY 
Degree: D.DES. 
Year: 1996 

Corporate Source/Institution: HARVARD UNIVERSITY (0084) 

Source: VOLUME 57/07-A OF DISSERTATION ABSTRACTS INTERNATIONAL. 

PAGE 2717. 210 PAGES 
Descriptors: INFORMATION SCIENCE ; MUSIC ; COMPUTER SCIENCE 
Descriptor Codes: 0723; 0413; 0984 

Computer games are a powerful paradigm for designing computational 
instruments for expression. Instruments share with games the need for 
multidimensional information handling, mental modeling techniques and a 
sense of wieldability . They should inherit those properties of games that 
aid a user in building a mental model of the way an artifact works. 
Specifically , a game model provides: (1) construction of a constrained 
universe with specific laws: narration of place; (2) user expectations of 
what happens next guided by context: narration of contextual moment; (3) 
virtually instant multimodal feedback to user^s actions: narration 
describing behavior. 

We examine narrative as a way to address the interaction between 
artifact and user. In particular we detail design parameters for audio in 
the multimodal realm, and how a narrative approach is applied at global, 
scenario, and micro levels in constructing auditory events. A narrative 
approach to audio design at the waveform level, using both digital signal 
processing and analog manipulation techniques, is described. 
Characteristic structures of sonic point of view, sonic place, sonic 
character, and sonic behavior are described, as well as remedies for design 
problems that arise from combining these. Our methods for designing audio 
through narrative work not only as a building technique but as a 
analytical tool that allows critical discussion prior to the stage of user 
testing. 

Four new computational instruments are described, with particular 
attention to the auditory elements of the design and implementation of 
each. We examine the commonalities between the instruments as well as the 
differing avenues each. explores . The Sonic Avatar combines three types of 
auditory display into an instrument for expressing personal motion in a 3D 

virtual environment. The Portal allows its user to play the air like a 
musical instrument: specifically , a percussive instrument with vocal 
qualities. The Health Care Chair uses new media technology to allow 
personal, home-based control over an increasingly complex medical data 
system. Animaurals is a realtime generation system for animation and 
sound, intrinsically multimodal in design. All instruments are designed for 
use by a single person; all are multimodal; all inherit different 
computational approaches to the interplay of information and expression. 
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p. 1410-18 
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Material Identity Number: E34 6-1 999-004 

Language: Korean Document Type: Journal Paper (JP) 

Treatment: Practical (P) ; Theoretical (T) 

Abstract: Facial expression plays an important role in human 
communications with people or machines. Our aim is to control an avatar by 
exploiting facial features so that the avatar can mimic the user's facial 
expression. Our approach consists of three main modules. The first tracks 
facial features and the second makes a model of a 3D avatar . The 
third transmits information from the feature tracking module to the 
avatar control module. We use the Gabor wavelet transform when extracting 
facial features . (15 Refs) 
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Identifiers: avatar control system; facial feature extraction; facial 
expression; human communications; 3D avatar; avatar control module; 
feature tracking module; Gabor wavelet transform 

Class Codes: B6135E (Image recognition); B0230 (Integral transforms); 
C1250M (Image recognition); C6180 (User interfaces); C5260B (Computer 
vision and image processing techniques); C1130 (Integral transforms) 

Copyright 1999, lEE 



15/5/18 (Item 8 from file: 2) 

DIALOG (R) File 2 : INSPEC 

(c) 2004 Institution of Electrical Engineers. All rts. reserv. 



5928388 INSPEC Abstract Number: C9807-7400-007 
Title: Digital humans in the simulated product life cycle 

Author(s): Miller, J.S. 

Author Affiliation: Deneb Robotics Inc., Auburn Hills, MI, USA 
Journal: HE Solutions p. 24-9 
Publisher: Inst. Ind. Eng, 
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Abstract: Digital or virtual humans are computer generated , 
graphically displayed entities that represent either imaginary characters 
or real humans. The former , commonly referred to as avatars , are used 
primarily in video games and the entertainment industry. The other 

digital humans, those of most interest to manufacturing companies, exist 
within a three dimensional ( 3D ) graphical simulation environment 
for the purpose of engineering evaluation. Though these digital humans may 
also vary in size , shape, and capability, they are intended to mirror 
actual human characteristics within a given population. This form of 
digital human can also be taught to perform the tasks required of its human 
counterpart. They are the virtual operators acting within simulated 
environments, manufacturing new vehicle components, assembling new 
aircraft, and maintaining the next generation of nuclear submarines and 
power plants. They are even training real operators how to perform future 
assignments. In conclusion, it is predicted that the use of human 
simulation will continue. The ability to predict human performance, with 
simulation of the various stages of a product's life cycle, will enable 
correct engineering decisions to be made earlier and at less cost than 
before. For lEs with an eye toward proactive participation in the product 
life cycle, digital humans may be the perfect partners. Putting a digital 
human to work may be the best way to ensure that a product is successful. 
(0 Refs) 
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Language: English Document Type: Conference Paper (PA) 
Treatment: Practical (P) 

Abstract: The IMPROV Project at NYU ' s Media Research Lab is building 
the technologies to produce distributed 3D virtual environments in which 
human-directed avatars and computer-controlled agents interact with 
each other in real-time, through a combination of procedural animation 
and behavioral scripting techniques developed in-house. We are also 
exploring multi-modal interaction paradigms combining traditional forms of 
input (keyboard and mouse) with speech and gesture recognition in 
conjunction with various forms of presentation, including 2D and 3D 
display. The system is intended to operate over local and wide area 
networks using standard internet protocols, enabling anyone with access to 
the World Wide Web to develop or participate in fully interactive, virtual 
experiences. (0 Refs) 
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Abstract: Twenty-five years after Logo's birth, there has been tremendous 
progress in programming language research and in computer-human interfaces. 
Programming languages exist now that are very expressive and mathematically 
very elegant and yet are difficult to learn and master. We believe the time 
is now ripe to attempt to repeat the success of the designers of Logo by 
child engineering one of these modern languages. Animation is much better 
suited for dealing with the dynamics of computer programs than static icons 
or diagrams. While there has been substantial progress in graphical user 
interfaces in the last twenty-five years, we chose to look not primarily at 
the desktop metaphor for ideas but instead at video games . Video 

games are typically more direct, more concrete, and easier to learn than 
other software. And more fun too. We have constructed a general-purpose 
concurrent programming system, ToonTalk, in which the source code is 

animated and the programming environment is a video game . Every 
abstract computational aspect is mapped into a concrete metaphor. For 
example, a computation is a city, an active object or agent is a house, 
birds carry messages between houses, a method or clause is a robot trained 
by the user and so on. The programmer controls a "programmer persona " in 
this video world to construct, run, debug and modify programs. We believe 
that ToonTalk is especially well suited for giving children the opportunity 
to build real programs in a manner that is easy to learn and fun to do. 
(8 Refs) 
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ABSTRACT: A scientist at the University of Surrey, England, has developed 
a digital photo booth that allows people to create 3-D computer images 
of themselves. This will let you send a computerized 3-D image , an 
avatar , of yourself into Internet chat rooms or into networked computer 
games . These images could be used to personalize e-mails or web sites 
and could be used as a form of identification on the Internet. However, 
the technique currently has some glitches that are being worked on. 
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Abstract (Basic) : WO 200190869 Al 

NOVELTY - User controllable images are generated in response to 
user inputs and are stored in a memory. The user controllable image 
is constructed such that the image is controlled to perform movements 
in response to input from user, instead of the user performing 
corresponding actual physical movements. The image movements are 
recorded and modified to represent idiosyncratic movements as 
controlled by the user and output to a display unit. 

USE - For operating processing system simulating any physical 
movement in any simulated environment e.g. for simulating physical 
movements like skating and hockey maneuvers, which is used for 
learning, pre-training and amusement in home and arcade through 
Internet and for industrial, commercial and sports application and 
computer video games for controlling motivity devices such as 
robots, avatars or other motiles to perform actual physical 
movements . 

ADVANTAGE - Enables providing the capacity to users to create 
user specific visual effects from user specific inputs. Enables 
developing personalized artificial intelligence in the form of 
stimulated memory, decision making and motor skills. 

DESCRIPTION OF DRAWING (S) - The figure shows the flow chart 
explaining the operation of processing system, 
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Abstract (Basic) : WO 200161447 Al 

NOVELTY ~ A computer receives performer movement information 
generated by a manual input device, based on manual manipulations and 
combines received information with create and stored character 
representation information to create character motion information. 
Another computer displays graphic character substantially 
synchronized to line manual manipulation, based on received character 
motion information. 

DETAILED DESCRIPTION - INDEPENDENT CLAIMS are also included for the 
fol lowing : 

(a) Computer system; 

(b) Computer readable medium; 

(c) Motion picture producing method; 

(d) Computer graphic character information creating and storing 

method 

USE - For controlling computer graphic characters and electro 
mechanically actuated puppet characters that are used in production 
of motion pictures in video game and web site. 

ADVANTAGE - Enables performer to create expression and 
performance for computer graphic character in a manner that is 
similar to a puppet performance, based on character motion 
information. Increases signal integrity by minimizing the number of 
interfaces and the number of signal forms . Performers are required to 
learn only a minimum number of software tools that are quickly and 
easily accessible. 

DESCRIPTION OF DRAWING (S) - The figure shows the flow diagram of 
computer graphic character performance method. 
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Abstract (Basic) : TW 266277 A 

A display list technology utilises an intentionally generated , 
location- specific vertical interrupt to implement a routine to modify 
or to alter existing graphical character elements. A second 
technique involves the definition of a small portion of the graphics 
map which is displayed differently than the balance of the graphics 
map because the stored priority bit is expressed in the defined area, 
but suppressed and replaced in all other areas. 

The third technique employs a virtual character element library 
to map the character elements appearing on the display, and 
recognizes available space within the video random access memory 
character element storage to provide a dynamic memory space. 

ADVANTAGE - Enhanced capability of video game system controller 
and data storage to provide graphic character element storage and 
process ing . 
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Abstract (Basic) : EP 661658 A 

The compact disk (CD) includes recorded data being an authored 
segment of data created from graphics and control information, and 
a personalised segment of data. A video game having one or more 
characters , and several personalised images (20, 22 and 24) are 
provided . 

The video game is read and executed. Selected personalised 
images are accessed and combined with one or more characters to 
form a personalised character in the game. The personalised 
character is then displayed (14) and used in the game. 

ADVANTAGE - Provides interactive entertainment system incorporating 
personalised images and utilises storage medium, Compact Disk, having 
integrated personalised data and generalised game data. 
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Abstract (Basic) : US 4517654 A 

The appts. comprises a device for storing image forming 
instructions at addressable locations and for supplying instructions 
from each location. A processor coupled to the device assembles a 
stationary plane image and a movable plane image during a display 
interval in accordance with the instructions input via a processor data 
bus, A video controller coupled to the processor generates vertical 
and horizontal display synchronisation signal outputs. 

A character generator includes several addressable character 
blocks and a second device has an input coupled to the processor for 
storing the assembled stationary plane image . A row subtracter 
coupled to the processor discretely shifts the movable plane image , 
in a vertical axis orce during each display interval. A device is 
coupled to the character generator for selecting between display of 
stationary and movable plane images and forming composite image 
to be displayed. 

USE - For video games . 
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ABSTRACT 

PURPOSE: To generate an easy-to-see character image on a game machine 
by registering character patterns in a plane 0 of a generator and then 
registering a character pattern which is shifted in direction and 
includes 0 as overlapping bits in a plane 1. 

CONSTITUTION: When the computer game machine displays characters on 
its video screen, the character patterns are registered in the plane 0 of 
a splite generator . Further, the same data with the plane 0 are described 
in the plane 1 so that bit patterns in the plane 0 are shifted right by one 
bit and down by one bit. In this case, the plane 0 is given priority for 
the bits overlapping with the plane 0 and the bits of the plane 1 are set 
to 0, thus processing the character patterns. Consequently, the plane 0 
is preferentially displayed as characters and the plane 1 is displayed as 
patterns for making the characters attractive. Therefore, the characters 
are seen in an embossed form and becomes easy to see. 
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Three-dimensional video edit display device stores specific image 
corresponding to input character attributes, while confirming display of 
character information 
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Abstract (Basic): JP 2002015339 A 

NOVELTY - A storage unit (4) stores character attributes input for 
extracting and storing a specific image, and stores the specific image 
data. A storage unit (9) stores the character attributes and scene data 
from the storage unit (4). An output image formation unit (8) generates 
3D video piled with character information based on image and 
character formation units (6,7) that generate video and character 
based on the stored information. 

USE - For editing and displaying three-dimensional video. 

ADVANTAGE - Since the display of three dimensional video and edit 
of character information are synchronized, three-dimensional video 
editing can be performed, efficiently. 

DESCRIPTION OF DRAWING (S) - The figure shows the block diagram of 
the three-dimensional video edit display device. (Drawing includes 
non-English language text) . 

Storage units (4,9) 
Image and character formation units (6,7) 

Output image formation unit (8) 
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Priority Applications (No Type Date) : GB 966129 A 19960322 
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JP 10003550 A 10 G06T-015/00 
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Abstract (Basic) : EP 797173 A 

The virtual environment navigation apparatus includes a data memory 
holding data defining the virtual environment, and a memory which 
stores data defining the appearance of an artefact in two or more 
poses. When a user is presented with an image of a virtual environment 
and enabled to move the image viewpoint location and direction within 
the environment, a control unit is provided for movable or animated 
characters or artifacts appearing within the environment. 

Variables, (distance, theta) which represent the distance between 
the viewpoint and the artefact, and the angle between what the camera 
is looking at, and the figure, are generated for determining the 
separation between the viewpoint and artefact and divergence of 
viewpoint direction. Changes in one or both of the variables initiates 
a controlled sequence (104. C/F) of poses adopted by all or part of the 
artefact . 

USE - Virtual environment navigation and interaction apparatus for 
interactive entertainment apparatus in which user navigates image of 
virtual environment. Provides computer- generated characters or 
features within virtual world with which user may wish to interact. 
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